Bayesian Additive Regression Trees & The Next Generation of Juvenile Justice Risk Assessments

RN Poulson University of Utah Final Defense, Spring 2019

Introduction

Recidivism Risk



Overall

  • Structured tools to estimate recidivism risk.

  • Statistical analysis of theoretically based factors.

  • Research shows outcomes are maximized.

Utah

  • Costs range from $11 to $1,224 per youth per day.

  • Risk assessment data shapes policies.

  • HB 239 \(\rightarrow\) 2 primary tools:
    • PSRA/PRA (Study 1)
    • DRAT (Study 2).

Literature


Study 1

  • Decades of research.

  • U.S. and worldwide.

  • Debate around methods (progressive vs. conventional).

Study 2

  • Experts recommend.

  • Adoption in many states.

  • Committee consensus.


Methods

BART

Basic Decision Tree
\(g(x;T,M)\)

BART


Posterior Estimation

\(p(T,M,Z|X,Y)\propto p(Y|Z)p(Z|X,T,M)[\Pi_j\Pi_i p(\mu_{ij}|T_j)p(T_j)]\)
Ultimate goal generate a large number of trees given the observed data to estimate \(p(T,M,Z|X,Y)\)
\(\Downarrow\)

Likelihood Function
\(p(Y|Z)p(Z|X,T,M)\)

Regularization Prior
\(p(\mu_{ij}|T_j)\) prior distribution of the terminal node parameters to \(\Uparrow\) probability \(E(Y|x)\) in \(y_{min}\) and \(y_{max}\).

\(p(T_j)\) prior on \(T_j\) tree & includes three considerations: tree size, selection of predictors, and selection of values for splitting rules.

Posterior Sampling
Large parameter space \(\Rightarrow\) intractable calculations \(\Rightarrow\) MCMC algorithm comprised of Gibbs sampler to sample from posterior \(\Rightarrow\) gravitate toward regions of high posterior probability.
\(\Downarrow\)

Bayesian inferences about the estimation of f(x), predictions of y, posterior uncertainty, and the marginal effect of one or more predictors on the response. Model free variable selection.

Random Forests


  • Bagging and random selection of features.

  • Bagging
    • Classification models are fit to a collection of bootstrap samples, \(\textbf{Z}^{*b}, b = 1,2,...,\textbf{B}\),
    • Resulting class votes are combined and the overall classification is then based on a majority vote.

  • Introduces random selection of input variables in tree-growing process.

  • Involves growing \(B\) trees, \({T(x;\theta{_b})}^{B}_1\) via bootstrapped data where a number of input variables, \(m\), randomly selected as candidates for tree splits.

Logistic Regression


  • Logistic regression modifies linear probability model with logistic function to bound probability estimates 0-1.

\(P(y=1|\textbf{x})=G(\beta_0+\beta_1x_1+...+\beta_kx_k)\)

where

\(G(z) = exp(z)/[1+exp(z)]\)


  • Fit using maximum likelihood. \(\boldsymbol{\hat\beta}\) is maximized by setting the derivatives \[\begin{equation}L(\boldsymbol\beta)=\Sigma^n_{i=1}l_i(\boldsymbol\beta)\end{equation}\] to zero and solving.

Samples



Study 1
  • Youths who received PSRA/PRA.
  • July 1, 2008 to June 30, 2014.
  • Cohort 1: n = 15,244.
  • Cohort 2: n = 4,587.
Study 2
  • Youths eligible for detention.
  • CY 2014 to 2016.
  • n = 5,793.

Study 1 Variables


Delinquency History

Age at 1st Offense
Felonies
Misdemeanors
Against Person Felonies
Weapons Offenses
Against Person Misdemeanors
Orders to Detention
Orders to JJS Custody
Escapes
Failure to Appear Warrants


Demographics

Gender


Social Situtation

School
Friends
Child Welfare
Runaway/Kicked Out
Household Members Criminal/Delinquency History
Compliance with Parental Rules
Substance Use
Suspected Victim of Abuse/Neglect
Mental Health


Outcome

Recidivism defined as adjudication on a new charge within one year

Study 2 Variables


Delinquency History

Prior Sex Offense
Pre Sex Offense
Prior Overall Referral
Pre Overall Referral
Prior Felony Referral
Pre Felony Referral
Prior Misdemeanor Referral
Pre Misdemeanor Referral
Prior Drugrelated Referral
Pre Drugrelated Referral
Prior Violent Referral
Pre Violent Referral
Prior Against A Person Referral
Pre Against A Person Referral
Prior Against Property Referral
Pre Against Property Referral
Prior Against Public Order Referral
Pre Against Public Order Referral
Prior Contempt
Pre Contempt


Demographics

Age at First Felony or Misdemeanor
Gender

System Contact

Prior Juvenile Justice Services
Prior Community Placement
Prior Secure Care
Prior Division of Child and Family Services
Prior Probation
Prior Order to Detention

Intake Offense

Severity
Against A Person
Against Property
Against Public Order

Outcome

Recidivism as defined as collection of a charge for a new offense within 100 days.

Procedures



Study 1
  1. Define training & test.
  2. Build three models.
  3. Classification threshold.
  4. Test performance.
Study 2
  1. Define training & test.
  2. Variable selection.
  3. Magnitude and direction.
  4. Classification threshold.
  5. Define tools.
  6. Test performance.

Results

Study 1


Study 1 Results, AUC
BART Logistic Regression Random Forests
PSRA - cohort 1
Felony or Misdemeanor 0.6945 0.6912 0.6634
Felony 0.7101 0.7113 0.6857
Misdemeanor 0.6903 0.6871 0.6584
PRA - cohort 2
Felony or Misdemeanor 0.6623* 0.6537 0.6445
Felony 0.6154 0.6113 0.6125
Misdemeanor 0.6514* 0.6436 0.6345
Note:
*Significantly greater than the logistic regression AUC

Study 1


Study 1 Results, Somers’ d
BART Logistic Regression Random Forests
PSRA - cohort 1
Felony or Misdemeanor 0.3329 0.3294 0.2942
Felony 0.3902 0.3788 0.3172
Misdemeanor 0.3236 0.3219 0.2867
PRA - cohort 2
Felony or Misdemeanor 0.2949 0.2816 0.2468
Felony 0.2033 0.1670 0.1997
Misdemeanor 0.2770 0.2670 0.2273

Study 2


DRAT Predictive Validity Results
BART Logistic Regression Random Forests
AUC 0.6933* 0.6734 0.6922*
Somers’ d 0.2984 0.2689 0.2815
Balanced Accuracy 0.6492 0.6344 0.6407
Note:
*Significantly greater than the logistic regression AUC

Discussion

Implications


Research
  • Support for tree-based approaches in research.
  • Why take the risk?
Policy
  • Update current risk assessment.
  • Support use of best available method.
  • Evaluate performance and update over time.
  • Support data system and quality.

Limitations & Future Research

  • Follow-up excludes adult system.
  • Risk assessments are imperfect.
  • Data quality.
  • Incorporate more items.
  • Expand scope (individual \(\rightarrow\) institution/system).
  • Misclassification costs.

References

References


Baird, C. (2009). A Question of Evidence: A Critique of Risk Assessment Models Used in the Justice System. Madison: National Council of Crime and Delinquency.

Baird, C., Heinz, R., & Bemus, B. (1979). The Wisonsin Case Classification Staff Deployment Project: Two-Year Follow-Up Report. Madison, WI: Wisconsin Division of Correnctions.

Barnoski, R. (2004). Assessing Risk for Re-Offense: Validating the Washington State Juvenile Court Assessment. Olympia: Washington State Institute for Public Policy.

Berk, R. (2012). Criminal Justice Forecasts of Risk: A Machine Learning Approach. New York: Springer.

Bitna, K., Merlo, A. V., & Benekos, P. J. (2013). Effective Correctional Intervention Programmes for Juveniles: Review and Synthesis of Meta-Analytic Evidence. 169-189.

Blattenberger, G., & Fowles, R. (1995). Road Closures: Combining Data and Expert Opinion. In C. Gatsonis, J. S. Hodges, R. E. Kass, & N. D. Singpurwalla, Case Studies in Bayesian Statistics, Volume II. New York: Springer.

Bleich, J., & Berk, R. (2015). Extensions and Applications of Ensemble-of-Trees Methods in Machine Learning. Doctoral Dissertation. ProQuest.

Bleich, J., Kapelener, A., George, E. I., & Jensen, S. T. (2014). Variable Selection for BART: An Application to Gene Regulation. The Annals of Applied Statistics, 1750-1781.

References


Bonta, J., & Andrews, D. A. (2017). The Psychology of Criminal Conduct (6th ed.). New York: Taylor & Francis.

Bonta, J., & Wormith, J. S. (2013). Applying the Risk-Need-Responsivity Principles to Offender Assessment. In L. A. Craig, T. A. Grannon, & D. Louise, What Works in Offender Rehabilitation (pp. 71-93). Malden: John Wiley & Sons.

Bonta, J., Rugge, T., Scott, T., Bourgen, G., & Yessine, A. (2008). Exploring the Black Box of Community Supervision. Journal of Offender Rehabilitation, 248-270.

Chipman, H. A., George, E. I., & McCulloch, R. E. (2010). BART: Bayesian Additive Regression Trees. The Annals of Applied Statistics, 266-298.

Chipman, H., & Rob, M. (n.d.). BayesTree: Bayesian additive regression trees. Retrieved from https://CRAN.R-project.org/package=BayesTree

Chipman, H., George, E., Lemp, J., & McCulloch, R. (2010). Bayesian Flexible Modeling of Trip Duration. Transportation Research Part B, 686-698.

DeWitt, J. R., Poulson, R. N., & Wetherly, E. (2016). The Utah Pre-Screen Risk Assessment (PSRA) and the Protective and Risk Assessment (PRA) Validation Study II. Salt Lake City: Utah Adminstrative Office of the Courts and Division of Juvenile Justice Services.

DeWitt, J., & Lizon, R. (2008). The Utah Pre-Screen Risk Assessment (PSRA) and the Protective and Risk Assessment (PRA) Validation Study. Salt Lake City, Utah: Utah Adminstrative Office of the Courts and Division of Juvenile Justice Services.

References


Dowden, C., & Andrews, D. A. (2004). The Importance of Staff Practice in Delivering Effective Correctional Treatment: A Meta-Analytic Review of Core Correctional Practice. nternational Journal of Offender Therapy and Comparative Criminology, 203-214.

Freedman, D. A. (2010). Statistical Models and Causal Inference: A Dialogue with the Social Sciences. (D. Collier, J. S. Sekhon, & P. B. Stark, Eds.) New York: Cambridge University Press.

Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. Annals of Statistics, 1189-1232.

Gatti, U., Tremblay, R. E., & Vitaro, F. (2009). Iatrogenic Effect of Juvenile Justice. Child Psychology and Psychiatry, 991-998.

Gendreau, P. (1996). Principles of Effective Intervention With Offenders. In A. Harland, Choosing Correctional Options That Work: Defining the Demand and Evaluating the Suppl (pp. 117-130). Thousand Oaks: Sage Publications.

Gendreau, P., French, S. A., & Taylor, A. (Revised 2002). What Works (What Doesn’t Work). Invited Submission to the International Community Corrections Monograph Series Project.

Goodkind, Sara. (2005). Gender-Specific Services in the Juvenile Justice System: A Critical Examination. Affilia-journal of Women and Social Work - AFFILIA J WOMEN SOC WORK. 20. 52-70. 10.1177/0886109904272061.

References


Grove, W. M., Zald, D. H., Lebow, S. B., Snitz, E. B., & Nelson, C. (2000). Clinical Versus Mechanical Prediction: A Meta-Analysis. Psychological Assessment, 19-30.

Hamilton, Z., Kigerl, A., Campagna, M., Barnoski, R., Lee, S., Van Wormer, J., & Block, L. (2016). Designed to Fit: The Development and Validation of the STRONG-R Recidivism Risk Assessment. Criminal Justice and Behavior, 230-263.

Harris, A., Evans, H., & Beckett, K. (2010). Drawing Blood for Stones: Legal Debt and Social Inequality in the Contemporary United States. American Journal of Sociology, 1753-1799.

Harris, A., Evans, H., & Beckett, K. (2010). Drawing Blood from Stones: Legal Debt and Social Inequality in the Contemporary United States. American Journal of Sociology, 1753-1799.

Harris, G. T., M., E. R., Quinsey, V. L., Lalumiere, M. L., Boer, D., & Lang, C. (2003). A Mulitsite Comparison of Actuarial Risk Instruments for Sex Offenders. Psychological Assessment.

Harris, G. T., Rice, M. E., Quinsey, V. L., Lalumiere, M. L., Boer, D., & Lang, C. (2003). A Mulitsite Comparison of Actuarial Risk Instruments for Sex Offenders. Psychological Assessment.

Hastie, T., Tibshirani, R., & Friedman, J. (2017). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer .

Hill, B. M. (1986). Some Subjective Bayesian Considerations in the Selection of Models. Econometric Reviews, 191-246.

References


Homan, B., & Ziedenberg, J. (2007). The Dangers of Detention: The Impact of Incarcerating Youth in Detention and Other Secure Facilities. Retrieved March 2016, from http://www.aecf.org/resources/2014-juvenile-detention-alternatives-initiative-progress-report/: http://www.justicepolicy.org/images/upload/06-11_rep_dangersofdetention_jj.pdf

Howell, J. C. (2003). Preventing & Reducing Juvenile Delinquency. Thousand Oaks: Sage Pulbications.

Juvenile Justice Oversight Committee. (n.d.). HB 239 - Juvenile Justice Amendments. Retrieved March 2018, from Utah Board of Juvenile Justice, Utah Commission on Criminal and Juvenile Justice: https://justice.utah.gov/Juvenile/hb239.html

Kapelener, A., & Bleich, J. (2016). bartMachine: Machine Learning with Bayesian Additive Regression Trees. Journal of Statistical Software, 1-40.

Kennealy, P. J., Skeem, J. L., Manchak, S. M., & Louden, J. E. (2012). Firm, Fair, and Caring Officer- Offender Relationships Protect Against Supervision Failure. Law and Human Behavior, 496-505.

KiDeuk, K., & Duwe, G. (2016). Improving the Performance of Risk Assessments. In F. Taxman, Handbook on Risk and Need Assessment: Theory and Practice (pp. 114-139). Routledge.

Ladenberger, N. A., & Lipsey, M. W. (2005). A Meta- Analysis of Factors Associated with Effective Treatment. Journal of Experimental Criminology.

References

Ladenberger, N. A., & Lipsey, M. W. (2005). The Positive Effects of Cognitive-Behavioral Programs for Criminal Offenders. Journal of Experimental Criminology, 451-476.

Latessa, E. J., & Lowenkamp, C. (2006). What Works in Reducing Recidivism. University of St. Thomas Law Journal, 521-535.

Latessa, E. J., & Lowenkamp, C. (2010). The Role of Offender Risk Assessment: A Policy Maker Guide. Victims & Offenders, 203-219.

Lipsey, M. W. (2009). The Primary Factors that Characterize Effective Interventions with Juvenile Offenders: A Meta-Analytic Review. Victims & Offenders, 124-147.

Lipsey, M. W., Chapman, G. L., & Nana, A. L. (2001). Cognitive-Behavioral Programs for Offenders. The Annals of the American Academy of Political and Social Science, 144-157.

Lowenkamp, C. T., Latessa, E. J., & Holsinger, A. M. (2006). The Risk Principle in Action: What Have We Learned from 13,676 Offenders and 97 Correctional Programs? Crime & Delinquency, 77-93.

Mendel, R. A. (2011). No Place for Kids: The Case for Reducing Juvenile Incarceration. Baltimore: Annie E. Casey Foundation. Retrieved March 2018, from http://www.aecf.org/m/resourcedoc/aecf-NoPlaceForKidsFullReport-2011.pdf

Mendel, R. A. (2014). Juvenile Detention Alternatives Initiative Progress Report. Baltimore: Annie E. Casey Foundation. Retrieved March 2018, from http://www.aecf.org/m/resourcedoc/aecf-2014JDAIProgressReport-2014.pdf

References

Molm, L. D. (1988). The Structure and Use of Power: A Comparison of Reward and Punishment Power. Social Psychology Quarterly, 108-122.

Nadolny, T. L. (2017, September). How Computers are Predicting Crime - and Potentially Impacting Your Future . Retrieved from Philly Daily News: http://www.philly.com/philly/news/crime/risk-assessment-tool-criminal-justice-reform-20170921.html

Patterson, E. G. (2008). Civil Contempt and the Indigent Child Support Obligor: The Silent Return of Debtor’s Prison. Cornell Journal of Law and Public Policy, 95-141.

Pearson, J. (2004). Building Debt While Doing Time: Child Support and Incarceration. Judges’ Journal, 5-11.

Quinsey, V. L., Harris, G. T., Rice, M. E., & Cormier, C. A. (2006). Violent Offenders: Appraising and Managing Risk (2nd ed.). Washington: American Psychological Association.

Rice, M. E., & Harris, G. T. (2005). Comparing Effect Sizes in Follow-up Studies: ROC Area, Cohen’s d, and r. American Psychology-Law Society, 615-620.

Rice, M. E., & Harris, G. T. (2005). Comparing Effect Sizes in Follow-Up Studies: ROC Area, Cohen’s D, and r. American Psychology-Law Society, 615-620.

References

Schmidt, P., & Witte, A. D. (1988). Predicting Recidivism Using Survival Models. New York: Springer-Verlag.

Schwalbe, C. S. (2007). Risk Assessment for Juvenile Justice: A Meta-Analysis. American Psychology-Law Society, 449-462.

Schwartz, I. M., York, P., Greenwald, M., Ramos-Hernandez, A., & Feeley, L. (2016). Using Predictive Analytics and Machine Learning to Improve the Accuracy and Performance of Juvenile Justice Risk Assessment Instruments. In F. Taxman, Handbook on Risk and Need Assessment : Theory and Practice, edited by Faye S. Taxman (pp. 140-163). Routledge.

Skeem, J. L., Louden, J. E., Polaschek, D., & Camp, J. (2007). Assessing Relationship Quality in Mandated Community Treatment: Blending Care With Control. Psychological Assessment, 397-410.

Sparapani, R. A., Logan, B. R., McColluch, R. E., & Laud, P. W. (2016). Nonparametric Survival Analysis Using Bayesian Additive Regression Trees (BART). Statistics in Medicine, 2741-2753.

Stansfield, R., & Williams, K. R. (2014). Predicting Family Violence Recidivism Using DVSI-R. Criminal Justice and Criminal Behavior, 163-180.

State of Utah. (2017). Utah State Legislature. Retrieved March 2018, from H.B. 239 Juvenile Justice Amendments: https://le.utah.gov/~2017/bills/hbillenr/HB0239.pdf

References

Steinhart, D. (2006). A Practice Guide to Juvenile Detention Reform. Baltimore: Annie E. Casey Foundation. Retrieved March 2018, from http://www.aecf.org/m/resourcedoc/aecf-juveniledetentionriskassessment1-2006.pdf

Taxman, F. S., & Dezember, A. (2016). The Value and Importance of Risk and Need Assessment (RNA). In F. Taxman, Corrections & Sentencing: An Overview of the Handbook Handbook on Risk and Need Assessment : Theory and Practice. Routledge.

Ting, M. H., Chu, C. M., Zeng, G., Li, D., & Chng, G. (2017). Predicting Recidivism Amoung Youth Offenders: Augmenting Professional Judgement with Machine Learning Algorithms. Journal of Social Work, 1-19.

Ven der Put, C. E., Stams, G. J., & Dekovic, M. (2014). Assessment in the Netherlands: The Developement of a New Scoring System. Assessment, 92-107.

Waldmann, P. (2016). Genome-wide prediction using Bayesian additive regression trees. Genetics Selection Evolution.

Image Credits

unsplash-logo

Priscilla Du Preez

Pedro Lastra

Patrick Hendry

Clem Onojeghuo

<Nelly Volkovich

Appendix

Study 1 Thresholds

Identification of Classification Threshold Example
(BART, PSRA Cohort)

Study 1

Number of Youths by Risk Level
Cohort 1 (left), Cohort 2 (right)

PDP and ICE

Partial Dependence Plot and ICE Curves Example
Prior Overall Referral Total, BART

Thresholds

Classification Threshold Example (BART)

Variable Selection

BART Variable Importance Test

Variable Selection

Random Forests Variable Importance Test

Variable Selection

Logistic Regression Variable Selection and Estimated Coefficients
Esimate Pr(>|z|)
Intercept -2.6628 0.0000
Intake Offense Against Property (Compared to Against Person) 0.4569 0.0130
Intake Offense Against Public Order (Compared to Against Person) 0.0960 0.6497
Intake Offense Felony 2 (Compared to Felony 1) 0.6433 0.0565
Intake Offense Felony 3 (Compared to Felony 1) 0.9910 0.0028
Intake Offense Misdemeanor A (Compared to Felony 1) 0.7394 0.0336
Intake Offense Misdemeanor B (Compared to Felony 1) 1.1129 0.0054
Prior Juvenile Justice Services Custody Flag -0.7108 0.0115
Prior Order to Detention Flag 0.5083 0.0101
Prior Misdemeanor Referral Total 0.0925 0.0000
Prior Felony Referral Total -0.0790 0.1136

Study 2

Number of Youths by Risk Level

End